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INTRODUCTION 

Unmanned Aerial Vehicle Definitions and History 

A Department of Defense (DoD) Website 

(http://www.defense.qov/specials/uav2002/uavpaqe01.html) defined Unmanned Aerial 
Vehicles (UAVs) as “powered aerial vehicles sustained in flight by aerodynamic lift over 
most of their flight path and guided without an onboard crew.” A Human Systems 
Integration (HSI) perspective of UAV design and operations is advocated in this report 
as the best approach to improve UAV effectiveness and to develop future 
semiautomated UAV systems. 

Research and development of UAVs has been ongoing for at least 50 years (Gossett, 
2004) and some sources claim that the U.S. military has been involved with UAVs since 
1917 (DoD, 2002). Operational UAV systems have proliferated in the past decade, 
leading to the common perception that UAVs are “new technology.” Approximately 200 
types of UAVs are currently in the U.S. military inventory and the number may rise to 
500 over the next five years (Garamone, 2002). They range from micro-UAVs (spanning 
six to nine inches) to High-Altitude Long-Endurance (HALE) UAVs such as the Global 
Hawk, which is approximately the same size as a fighter aircraft, operates at altitudes of 
approximately 60,000 feet, and has an endurance of nearly two days. 

UAVs can provide the following benefits in DoD applications: 

• Eliminate the risk to ground troops and aircrew who would otherwise be 
performing the mission; 

• Enhance aerodynamic performance over manned flight due to lighter weight and 
freedom from human G-tolerance constraints; and 

• Reduce cost. 

The U.S. military is committed to rapid development and deployment of unmanned 
systems as evidenced by the following goals stated by the Senate Armed Services 
Committee: 


1 


Human Systems Integration and Automation Issues in SUAVs 


It shall be a goal of the Armed Forces to achieve the fielding of 
unmanned, remotely controlled technology such that: 

(1) by 2010, one third of the operational deep strike aircraft of the Armed 
Forces are unmanned; and 

(2) by 201 5, one third of the operational ground combat vehicles of the 
Armed Forces are unmanned (Senate Armed Services Committee, 2000). 


An overview of UAV military operations and development challenges can be found in the 
document, “DoD UAV Roadmap 2002” ( http://www.acq.osd.mil/usd/uav roadmap.pdf ). 
There has been controversy about the basis for classifying or categorizing UAVs. Some 
of the suggested criteria are airframe size, mission, support requirements, and cost. 
Airframe size is the most commonly used criterion. Small UAVs (SUAVs) are not 
distinguished by unique function, mission relevance, or because they are less 
expensive. The relevance of SUAVs is based on the operational impact of their 
logistics — they offer greater flexibility in operational employment compared to larger, 
more complex UAV systems (DoD, 2002). The definition of “Small UAV” given in the 
DoD UAV Roadmap 2002 is: 


a. For UAVs designed to be employed by themselves - any UAV system 
where all system components (i.e., air vehicles, ground control/user 
interface element, and communication equipment) are fully transportable 
by foot-mobile troops 

b. For UAVs designed to be employed from larger aircraft (manned or 
unmanned) - any UAV system where the air vehicle can be loaded on the 
larger aircraft without the use of mechanical loaders (i.e., two-man lift, 
etc.) 


The U.S. Army, Navy, Marine Corps, and Special Operations Forces have invested in 
the development of SUAVs to provide tactical imagery to small units (battalion and 
below) for reconnaissance, surveillance, target acquisition (RSTA), and battle damage 
assessment (BDA). Two U.S. Army UAV programs, the Hunter and the Shadow, have 
been deployed successfully in the Persian Gulf, Bosnia, Afghanistan, Iraq, and other 
conflict locations (DoD, 2002). 


The Army’s Future Combat System (FCS) initiative identifies a requirement for two 
classes of SUAVs: 


2 


Human Systems Integration and Automation Issues in SUAVs 


• Class I = Backpackable; and 

• Class II = Vehicle/Man Transportable. 

In 2003, the Army Strategic Planning Board designated the SUAV as an “Urgent 
Wartime Requirement.” The Board projected that SUAVs will be used in direct support 
of the Global War on Terrorism with immediate application to Army forces engaged in 
Operation Iraqi Freedom and Operation Enduring Freedom. 

UAV research, development, and acquisition programs in the DoD have been sporadic 
and haphazard over the years and, perhaps as a consequence, the current technologies 
suffer from high loss rates and require substantial manpower to operate and maintain. 
One analysis concluded that the U.S. has had a “three-decade-long history of poor 
outcomes in unmanned aerial vehicle development efforts” (Leonard and Drezner, 

2002). However, the U.S. is entering a new era in which the unbridled engineering 
innovation that has characterized UAV design and development, is likely to give way to 
more systematic, systems engineering approaches. 

Report Objectives 

The goal of this report is to identify HSI and automation issues that contribute to 
improved effectiveness, efficiency, and risk management in the operation of U.S. military 
SUAVs. To that end, the following three objectives were defined: 

1 ) Summarize current HSI issues relevant to SUAV operations; 

2) Identify areas where short-term HSI improvements might yield gains in 
effectiveness; and 

3) Identify research, design concepts, and technologies in automation and related 
disciplines that are applicable to future SUAVs. 

SUAVs hold great promise and will evolve into valuable tactical assets. Any issues or 
problems identified in this report are intended to hasten the improvement cycle. 
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The present report is based on three primary sources of information: 

1) Participation in a series of UAV experiments and field trials at Camp Roberts, 
California, conducted by the Naval Postgraduate School under the direction of 
Dr. David Netzer; 

2) Review of the literature in UAVs, semiautonomous systems, robotics, and related 
areas, with a focus on HSI and evolving technology; and 

3) Attendance at the Association for Unmanned Vehicle Systems International 
(AUVSI) 2003 conference and flight demonstrations at Webster Field, Maryland. 

SUAV Overview 

This paper will focus on U.S. military SUAVs, as opposed to HALE UAVs or Medium 
Altitude and Endurance (MAE) UAVs, such as the Global Hawk or the Predator. High 
altitude generally refers to above 50,000 feet and long endurance to 24 hours or more. 
By contrast, current SUAVs tend to operate at less than 5,000 feet and their endurance 
is on the order of several hours. Examples of current SUAVs are the TERN, Silver Fox, 
Swift, Pointer, Raven, and Dragon Eye. The Hunter and the Shadow, by contrast, are 
larger and usually are called “Tactical” UAVs (TUAVs). In this report, the term SUAV is 
used routinely, but the content of this report is equally appropriate for both SUAV and 
TUAV design concepts and operation. 

SUAV missions often are described as “what’s over the next hill?” Related questions 
might be, “where are they?” or “how many are there?” SUAVs enable small, ground- 
based military units to perform intelligence, surveillance, and reconnaissance (ISR) 
missions, reducing the risk to mounted or dismounted troops who would otherwise 
perform the ISR mission. The objective is to extend the “eyes” of the unit over a greater 
range, more quickly, and with reduced risk to human life. According to the DoD 
Roadmap (2002), one of the five “historically validated UAV roles is small unit asset for 
over-the-hill reconnaissance.” SUAVs will be deployable at or near the front lines, at the 
company or platoon level, and “will provide the commander with what amounts to a pair 
of flying binoculars” (DoD, 2002). 

SUAVs provide electro-optical (EO) or infrared (IR) sensor data leading to the detection, 
classification, and identification of vehicles, people, and other tactically relevant objects. 
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From the HSI perspective, it is important to note that SUAVs do not detect, classify, or 
identify anything — people do. The SUAV provides the video or IR imagery or other data 
that enables properly trained observers to perform the detection, classification, and 
identification functions. 

UAV System Example 

The basic structure of a typical UAV system may be viewed as an aircraft with the crew 
physically remote from the vehicle. The upside of the remote crew is reduced human 
risk, the downside is reduced sensory input about the status of the aircraft (e.g., no 
noise, vibration, vestibular, or proprioceptive input) and reduced visual information 
regarding terrain, weather, air traffic, and threats. 

A typical UAV system has a minimum of two operators, a pilot or Air Vehicle Operator 
(AVO) and a sensor operator or Mission Payload Operator (MPO). As shown in Figure 
1, these operators work in the Ground Control Station (GCS), which typically is installed 
in a stationary High Mobility Multipurpose Wheeled Vehicle (HMMWV; or“Humvee”). 



Figure 1. Ground Control Station. 

A schematic diagram of a typical UAV system is given in Figure 2, including a single 
UAV, a GCS with two operators, and an image analyst, who represents either a unit 
commander or a “customer” of the UAV payload data. 

The links depict communication (T/R = Transmitter; R/V = Receiver) between the 
corresponding links. For the purposes of this report, the methods used to implement 
these communications are not critical. 
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Figure 2. Schematic diagram of a UAV system. 
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PROBLEM DEFINITION 

The current performance of SUAVs has room for improvement, reflecting a technology 
that is still under development. Some of the challenges that must be faced are that 
SUAVs: 

• are manpower intensive; 

• have a large “footprint”; 

• need to improve visual data display; and 

• have a high loss rate. 

Manpower Intensive 

Current Army practice is to assign up to 24 people and three UAVs per UAV system. 
Typically, only one of the three UAVs is airborne at a given time, yielding a “worst case” 
24:1 ratio of humans to UAVs. Current UAVs are controlled from the GCS with two 
operators and often two additional people — an image analyst and a unit leader. Thus, 
the “Human-to-UAV” ratio could be characterized as 4:1, or 3:1 , or, best case, 2:1 . 

The Defense Advanced Research Project Agency (DARPA) has attempted to push the 
limits on human control of automata, including UAVs. The objective of the DARPA 
Mixed-Initiative Control of Automa-teams (MICA) program was to design a system that 
would enable 30 humans to control 300 entities, or a ratio of 1 :10. That would represent 
somewhere between a 20- to 240-fold improvement over current SUAV manning 
requirements. This challenge has been called “inverting the control ratio” 

(Johnson, 2003). 

Manpower costs are the largest driver of system life cycle costs. Improving the ratio of 
humans to UAVs will enable major reductions in the life cycle cost of a UAV system. 
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SUAV Footprint 

The term “footprint” is used to convey both physical and logistical footprint. Future 
concepts of Small or Micro-UAVs envision air vehicles small enough and light enough to 
be carried in a backpack, hand-launched, and managed by one person. The current 
reality is that a manned SUAV system includes the following components: 

• GCS, housed and transported in a HMMWV; 

• Antennas, generators, and communications gear; 

• Remote Video Terminal; 

• 24 people; 

• 3 UAVs (4 for Hunter); and 

• Consumable spares. 

The physical and logistical footprint is large and unduly cumbersome to achieve the 
benefit of one airborne UAV. The challenge is to develop future SUAV systems that are 
smaller, lighter, more agile, and require less logistic support. 

Visual Data Display and Search Effectiveness 

Time for Visual Target Detection . Visual search is a demanding, time-consuming task 
that requires multilayered analysis of human perceptual and cognitive systems 
(Neisser, 1967). In one target detection experiment (Itti, Gold, and Koch, 2001), the 
participants were instructed to detect a target in a natural scene photograph; a task 
similar to UAV search task. The average time required for target detection was 
2.8 seconds. When UAV operations lead to a high rate of video flow, the target may not 
remain visible on the monitor for 2.8 seconds. Operational factors that contribute to this 
visual search problem are unstabilized imagery, narrow field of view (FOV), 
low visibility, communication drop outs, and high rate of visual flow (high speed and/or 
low altitude). 

Figure 3 is a representation of the time available to the image analyst to detect a target 
from the visual scene as a function of altitude and speed. The assumptions underlying 
the model are: 
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• The UAV is flying at constant speed over ground (in [km/hour]) and height (in [ft]). 
Typical air speed for existing SUAVs is in the 80 to 100 km/hr range. 

• The camera is stabilized. 

• The target is stationary. 

• There are no lags, or other errors. 

• The FOV (30°) of the camera corresponds to a typical value found in existing 
SUAVs. 

Under these assumptions, the plots in Figure 3 depict the time available for detection by 
the image analyst. Clearly, low speed and high altitude provide more time for the 
analyst. If the 2.8-second limit is applied to this theoretical plot, we find that only 
altitudes above 500 feet are likely to result in target detection. 



—♦—200 [ft] 
—■—500 [ft] 
— A — 800 [ft] 
1100 [ft] 
— * — 1300 [ft] 
—•—1500 [ft] 


Figure 3. Available time to detect a target. 

Visual Data Display Challenges . The demand for “Tactical”-size UAV services in Iraq 
and Afghanistan indicates that commanders in the field place a high value on the tactical 
data provided by the TUAVs. However, observation of SUAV operations at 
Camp Roberts and discussions with operators indicate that improved imagery and 
image stabilization are needed to improve the ISR capability of SUAVs. 
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Several issues were observed to influence ISR effectiveness during SUAV search 
operations at Camp Roberts. The objective of one set of exercises was to detect and 
classify three types of military ground vehicles in an area approximately 2x3 km. The 
terrain was hilly with clusters of oak trees and occasional dirt roads. Two types of 
SUAVs were engaged in the search, with a surrogate HALE UAV, the Pelican, operated 
by the Center for Interdisciplinary Remotely Piloted Aircraft Studies (CIRPAS). The 
three Army ground vehicles followed a script for periods of movement followed by either 
a stop with no concealment or partial concealment under a tree. One of the SUAVs had 
a fixed “lipstick” type of camera and the other SUAV had a larger, pan/tilt/zoom (PTZ) 
camera. 

Based on our observations, camera type had a large influence on ISR effectiveness. 

The searches conducted with a fixed camera were unproductive. If the altitude were 
high enough (approximately 2,000 ft above ground level) to support a reasonable field of 
view, then the size (retinal angle subtended by the displayed target) and resolution of the 
target/object images was insufficient to enable detection by the human observer, given 
the “bounce” of the imagery caused by mild turbulence. If the SUAV altitude were low 
enough, then the field of view (sweep width) was small, the optical flow rate was high, 
and the time available for detection was low. The image analyst’s task was difficult at 
low altitude due to the combination of the high rate of optical flow and the “bounce” of 
the imagery. 

The imagery bounce was considerably more than a small vibration. It was observed to 
be a magnitude of nearly one video frame, oscillating irregularly at a frequency of 
approximately 2-4 Hz. For example, if an object were at the top of the monitor image, it 
would go near, or off, the bottom of the monitor image several times per second. A 
quantitative analysis of this effect is recommended, leading to recommendations for 
imagery stabilization. 

Searches conducted with a stabilized PTZ camera operated by a MPO were far superior 
to the fixed-camera system. Imagery from the Navy’s TERN UAV, based on a PTZ 
camera, but nonoptimal optics, was adequate for the detection task. The surrogate 
HALE UAV was not used as a primary search sensor, but it provided imagery of 
extraordinary quality, stability, and resolution. 
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In the opinion of this observer, the fixed “lipstick” camera was insufficient to support 
ISR operations. It was extremely difficult to detect a military vehicle parked in the open 
at a known crossroads and nearly impossible to detect a vehicle partially obscured under 
a tree. Perhaps this type of SUAV search system could provide useful imagery under 
constrained conditions, such as very smooth air and open areas with no trees or other 
concealment. In the Camp Roberts exercises, systems with a shock-mounted, 

PTZ camera provided far better imagery for the detection task, but that performance 
comes with a weight penalty. Some research has been done on the development of an 
intelligent, semiautonomous interface for stabilization of UAV camera images 
(Korteling and van der Borg, 1997). The relationship between image quality, probability 
of detection, and the weight penalties that come with a shock-mounted, PTZ camera 
need to be quantified to support future SUAV design decisions. 

One HSI challenge is to define the detection system requirements by working 
“backwards” from the observer to the sensor. Existing methods for characterizing 
sensor resolution, like the National Imagery Interpretability Rating Scales (NIIRS) or 
Johnson criteria (Leachtenaur and Driggers, 2001), need to be extended to characterize 
the performance of trained human analysts monitoring a streaming video image. This 
analysis is needed not only for static optical resolution, but for dynamic issues such as 
optical flow, vibration, and image bounce caused by turbulence and the aerodynamic 
response of the air vehicle. 

Additionally, research is needed to determine the probability of detection for streaming 
video imagery compared to a series of still images. The frequency and duration of still 
images are variables that need to be tested. One design option is to allow the operator 
(image analyst) to select the image iteration rate. Tests are needed to determine 
whether operator detection performance with unstabilized video imagery is better when 
viewing a series of static frames. Research on this issue should be done with both static 
and moving targets. It is possible that static images may be better for stationary targets, 
while streaming video could have an advantage for detecting moving targets. 

Experts in visual perception should analyze the engineering characteristics of UAV video 
displays to optimize visual performance and pattern recognition. Engineering progress 
in reducing the size and weight of UAVs will be beneficial only if the video product is 
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useful, i.e., stabilized and with sufficient resolution and contrast for the analyst/observer 
to achieve acceptable detection/classification performance. 

UAV Loss Rate 

The loss rate for UAVs is several times that for manned aircraft. The reliability and 
sustainability of UAVs establishes the basis for their affordability and their mission 
availability (DoD, 2002). Accurate data on UAV reliability are difficult to obtain and that 
seems particularly true for SUAVs. There is no standardized database for UAV mishap 
reporting and there are a large number of UAV systems developed by a large number of 
manufacturers and operated by different branches of the military. A definition of UAV 
reliability comprising four metrics has been proposed (DoD, 2002): 


1 ) Mishap Rate is the number of accidents occurring per 1 00,000 flight hours. 

2) Mean Time Between Failure (MTBF) is the ratio of hours flown to the number of 
maintenance-related cancellations encountered (expressed in hours). 

3) Availability is the number of times a given aircraft type is able to perform its 
missions compared to the number of times it is tasked to do so, expressed as a 
percentage (describes the performance of a system while on standby). 

4) Reliability is 100 minus the percentage of times a launched mission is either 
canceled before takeoff or aborted during flight due to maintenance issues, 
expressed as a percentage (describes the performance of a system while in 
operation). 


Table 1. Four UAV Reliability Metrics (derived from DoD, 2002). 


System 

Mishap Rate* 
per 100,000 hrs. 

MTBF 

(hrs.) 

Availability 

Reliability 

Predator 

32 

44 

67% 

82% 

Pioneer 

334 

14 

76% 

86% 

Hunter 

55 

11 

98% 

82% 


*Class A mishaps, resulting in the loss of the aircraft, a death, or more than 
$1 ,000,000 in damage; for two model series (RQ-1A & -IB; RQ-2A & -2B; RQ-5 
pre-/post-1996). 


No data were reported for small UAVs, given that the Hunter is usually classified as a 
“Tactical” UAV. The poor MTBF results, however, are consistent with our observations 
at the Naval Postgraduate School experiments at Camp Roberts, where four different 
types of UAVs were flown on multiple sorties on multiple days. While no official record 
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of UAV losses is available, it was clear that mishaps were frequent, on the order of one 
every other day. Based on an estimate of 8-12 flight hours per day, that equates to 
approximately one mishap per 16-24 flight hours. Discussions with one group of 
experienced, military SUAV operators at Camp Roberts led to their estimate that, for 
their type of SUAV, the average airframe life duration is approximately 20 flight hours. 
This figure is consistent with the MTBF data reported by DoD (2002) in Table 1 . It is 
also consistent with the report that NATO lost 20 to 30 UAVs during the 78-day Kosovo 
air campaign (Mouloua, Gilson, and Hancock, 2003). 

In the series of experiments coordinated by the Naval Postgraduate School at 
Camp Roberts, the causes and contributing factors of mishaps often were uncertain. In 
some cases, however, the cause was obvious, such as running out of fuel. Most, if not 
all, current UAVs have no fuel gauge. The operators estimate the flight duration and 
recover the UAV according to that schedule. When operational requirements place a 
high value on a few more minutes on station, the fuel reserve may disappear, especially 
when high ambient temperature has increased the fuel burn rate. As a side note, the 
tarmac temperature at Camp Roberts on one day of operations was 146° F, probably 
similar to the temperatures experienced in Iraq. 

Other SUAV losses observed at Camp Roberts were due to delamination of the wing 
(perhaps from long-term storage in very hot, humid conditions in the southeastern U.S.) 
or to an infrequent, but irksome, software bug that shuts off the engine. 

A preliminary analysis of 48 UAV mishaps was reported by the DoD’s Human-Systems 
Information Analysis Center (HSIAC), based on data from the Air Force and Army Safety 
Centers (Rogers, Palmer, Chitwood, and Hover, 2004). This analysis included 10 years 
of data for Class A and Class B mishaps, defined as follows: 

Class A Mishap : Damage costs of $1 ,000,000 or more and/or destruction of 
an aircraft, missile or spacecraft and/or fatality or permanent total disability of 
personnel. 

Class B Mishap : Damage costs of between $200,000 and $1 ,000,000 
and/or permanent partial disability and/or three or more people hospitalized 
as inpatients. 
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Of the 48 mishaps, approximately one-third (15) were attributed to mechanical failure 
and, according to the judgments of Rodgers et al. (2004), all of the remaining 33 
mishaps involved some form of human-systems issues. The criteria for these judgments 
are unclear and unspecified in the report. 

The cost of the 48 mishaps was nearly $14 million for the Army and $177 million for the 
Air Force, with the overall average cost per mishap close to $4 million. A breakout by 
UAV type for the 48 mishaps is as follows: Predator 32%, Hunter 19%, Shadow 17%, 
Global Hawk 8%, others 24%. 

These data do not provide insight into SUAV loss rates because they will be less than 
the “Class B” definition of $200,000. While cost, fatalities, and injuries are the traditional 
measures of “loss” for manned aircraft mishaps, and may be appropriate for large UAVs, 
they may not be the appropriate criteria for SUAVs. SUAV mishaps are not terribly 
expensive, nor are they fatal, but they compromise ISR effectiveness and add to logistic 
burdens. When an SUAV is engaged in a reconnaissance mission, search time and 
coverage are lost if it goes down because it takes time to prepare and launch another 
SUAV to replace it. During that time, the target or object of the search/tracking mission, 
such as military vehicles or mobile launchers, may have moved to cover. If SUAV loss 
rates are high, a given unit may not have sufficient spares to provide SUAV ISR 
coverage. 

The HSIAC preliminary analysis of Rodgers, et al. (2004) suggests that two-thirds of 
SUAV mishaps involve human systems integration issues. This estimate is consistent 
with historical estimates of “human error” as a causal or contributing factor, which 
typically are in the range of 60%-80% of all accidents in ground transportation, aviation, 
and industry (Perrow, 1999). Given the current human-in-the-loop design of UAV control 
systems, we have no reason to believe that SUAV operations will be different. 

A study of the role of human and organizational factors in Army UAV accidents was 
reported by Manning, Rash, LeDuc, Koback, and McKeon (2004). The authors identified 
56 UAV accidents from the U.S. Army Safety Center’s database during the period 
FY95-FY03. The Army traditionally identifies three basic causes of accidents — human, 
materiel, and environmental factors. This report summarizes categories of 
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human-causal factors such as workload, fatigue, SA, training, crew coordination, and 
ergonomic design. Human error was found to play a role in approximately one-third 
(32%) of the reported accidents. Two methods of accident analysis were employed and 
both identified individual unsafe acts or failures as the most common human-related 
causal factor in UAV accidents. There are at least two important conclusions from the 
Manning et al. (2004) analysis: 

1) human-causal factors play a role in a substantial proportion (one-third) of UAV 
accidents; and 

2) the Army’s current accident reporting method does not support accurate capture 
of human error accident data. 

A systematic basis for tracking SUAV losses is needed. Accurate data on SUAV 
mishaps are needed to support analyses of causal and contributing factors so that 
improvements in reliability can be achieved. 

There are two challenges here: 

1) Improve SUAV reliability (the HSI specialists should focus on the subset of 
mishaps involving human operators or maintainers); and 

2) Develop a standardized reporting system and database for UAV reliability 
(including SUAVs) across all services. 

Acquisition methods for some SUAVs seem to have circumvented the traditional 
requirements for reliability analyses and Government acceptance testing, which may 
contribute to the mishaps attributable to hardware and software problems. 
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HSI ISSUES FROM NEAR-TERM AND LONG-TERM PERSPECTIVES 

Throughout the history of UAV development there has been little or no attention paid to 
applying “best practices” of HSI and user-centered design. But, UAV programs have 
matured to the point that haphazard user interface design is no longer acceptable and 
should not be allowed to contribute to high mishap rates and reduced system 
effectiveness. 

The following categories of HSI issues apply to UAV operations, both now and in the 
future: 


• Human Roles, Responsibilities and Level of Automation; 

• Command and Control; Concept of Operations; 

• Manning, Selection, Training, and Fatigue; 

• Difficult Operational Environments; 

• Procedures and Job Performance Aids; and 

• Moving Control Platforms. 

How can HSI improve SUAV operations? One way to address this question is to 
consider two different perspectives: near-term improvements that require no new 
technologies and longer-term improvements, such as semiautomated systems, that will 
require time and research and development investment. 


Near-Term Perspective 

Human Roles, Responsibilities, and Level of Automation 

The development of current SUAV systems has focused on the fundamental enabling 
technologies such as aeronautical design, sensors, and communication links with the 
air vehicle. The simplest SUAVs are essentially hobbyist model planes with a small 
camera installed. The human role in this type of system is to manually operate a 
remotely piloted vehicle or teleoperated system. 

Morphew (2003) interviewed a number of TUAV operators (AVOs and MPOs ) and found 
that the most difficult tasks were considered to be: 
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• Launch and landing; 

• Identifying and reacting in emergencies; and 

• Maintaining situational awareness (SA). 

Technologies are under development to support UAV operators in accomplishing these 
tasks. Several flight management systems have been developed and are currently in 
use. For example, Piccolo Plus by Cloud Cap is a “user programmable autopilot for 
UAVs.” This technology provides basic automation functions such as altitude hold and 
waypoint following. These automation features reduce the operator workload by taking 
him/her out of the continuous manual control loop. They also change the role of the 
human operator from direct control to a hybrid of direct and supervisory control. 

Further development is needed to provide a mission management system that 
comprises premission planning and dynamic replanning during a mission. A mission 
management system for SUAVs might contain the following features: 

• Preprogram, reprogram, and manage the flight profile; 

• Automated takeoff and landing; 

• Preprogram, reprogram, and manage the sensor payload; 

• Decision aiding for emergencies and degraded mode operations; and 

• Supplemental displays to support SA. 

This type of mission management capability exists in U.S. military systems — no new 
technology is necessary. Worthwhile objectives of a mission management system are to 
reduce the mishap rate, shrink the 24-person SUAV unit, and enable simultaneous 
control of multiple UAVs. 

The capability to manage and revise the flight profile and sensor payload is essential 
because, according to experienced military operators from Fort Huachuca and 
Patuxent River, the majority of all SUAV missions require revision of the mission plan 
during the mission. The operator must be afforded the capability to change parameters 
quickly and easily via an excellent user interface. Reverting to manual control should be 
an available option, but revising the flight profile should be possible without going 
manual. 
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In the future, the role of the human will expand to include control of multiple SUAVs and 
control from a moving vehicle (ground or air). These capabilities and role changes must 
be accompanied by changes in the user interfaces, training, and operating procedures. 
The “integration” of HSI is a central concept in evolving the capability of SUAV systems. 
A challenge for DoD acquisition is to manage this evolution, rather than to acquire new 
hardware/software, then subsequently deal with the inevitable problems that accompany 
post hoc development of operating concepts, procedures, training, and user interface 
functionality. 

Command and Control and Operations 

SUAV operators need to know where the sensor is pointing. This capability is not 
provided in most current SUAVs. The coordinates of the center of the sensor field of 
view are essential, for example, when an image analyst requests a more thorough 
investigation of a possible contact. Having Global Positioning System (GPS) data on the 
location of the air vehicle is not sufficient. The coordinates of the sensor view are 
necessary. A geographic tactical display is needed to show a historical track of the 
sensor coverage. This display mode would make gaps in coverage immediately evident 
to the MPO and image analysts. Ideally, the sensor history track would have a decay 
function selectable by the operator. 

System response time and the temporal aspect of team communications. During 
search operations at Camp Roberts, time lags were observed between an image analyst 
communicating “possible contact” and the coordinated response from the AVO and the 
MPO. This problem was exacerbated when the operators did not have information 
about the coordinates of the possible contact. The time lags can be characterized as an 
inside- and outside-loop control system. The AVO and MPO have inside-loop control of 
the UAV and the sensors. If they are using autopilot features, then they must reacquire 
manual control and take appropriate action. Time is required for the image analyst to 
assess the video data, perceive, decide, and communicate the possible contact. Then, 
time is required for the operators to respond. There are several possible solutions to this 
problem. The analyst, whether local or remote, could be given a manual, rather than 
verbal, response option that would signify “possible contact” and capture the relevant 
coordinates. Another approach is team training. The search team can practice the 
specific communication and performance necessary to minimize response times. 
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ISR search pattern analysis and codification in a “search template” tool. SUAV 
operators currently define a search pattern by manually entering data for a series of 
waypoints. This process is time consuming, tedious, and may not reflect analyses of 
alternative search patterns. The field of Operations Research can develop search and 
detection models that will estimate the probability of detection for various search 
patterns. A software tool could be provided to SUAV operators that would support 
optimal search patterns while eliminating the requirement to enter every waypoint. With 
the appropriate sensor performance data loaded into the software tool, operational 
variables such as altitude, sensor type and FOV, type of target, type of terrain, 
temperature, and winds could be included in a “Search Template” software tool. The 
operator would enter points defining the bounds of the search area, and then use the 
Search Template tool to define the UAV track that provides optimal coverage. 

Operations analysis leading to recommendations for system improvements. Most 
current SUAV systems were developed and fielded without the benefit of systems 
engineering and analysis. A thorough analysis of current SUAV operations, procedures, 
and training is recommended to determine ways to improve SUAV system effectiveness. 
A composite team of experts in aeronautics, sensors, command and control, 
communications, software, tactics and human factors/HSI would be ideally suited to 
accomplish these analyses. 

Manning, Selection, Training, and Fatigue 

UAVs are not unmanned systems. People operate UAVs and how the people are 
selected, trained, and scheduled contributes to system effectiveness. 

The manning criteria for SUAV operations should be reviewed periodically as the 
systems evolve. At the present time, operation in the National Air Space is unlikely for 
SUAVs and the Army’s current 96U MOS appears to be sufficient for Army SUAV 
operations. As the technology changes and the operational concepts change, both the 
number of personnel in a unit and the entry qualifications will need to be reevaluated 
periodically. 

An enlightening study would be to determine the probability of detection and the time to 
detection for the top 20% and the bottom 20% of operator skill and proficiency. 


19 



Human Systems Integration and Automation Issues in SUAVs 


Observation of SUAV operations at Camp Roberts indicated substantial differences in 
operator skill. The expert operator and team are much more likely to be successful. The 
challenge is to define the selection criteria and continually update them as the job and 
the pool of candidates evolve. 

Strong training programs produce excellent operators. Our unstructured interviews with 
operators indicated that the Army has a strong training program for UAV operators at 
Fort Huachuca. The architecture of UAV systems is conducive to embedded training in 
GCS design. Mission rehearsal training also is feasible, given that a database of the 
operating terrain and expected objects of interest are available and the setup tools are 
provided to establish meaningful scenarios. 

Fatigue, sleep, and circadian rhythm have strong effects on human performance 
(Miller, Nguyen, Sanchez, and Miller, 2003). Watch schedules for UAV operators are 
particularly important when engaged in day and night, 24-hour missions. Recent data 
indicates that a human operating on no sleep in a 24-hour period exhibits signal 
detection performance equivalent to someone with a 0.08 blood alcohol level, the legal 
limit in most states (Doheny, 2004). Manning decisions should ensure that a sufficient 
number of qualified and trained personnel is available to man a watch schedule. 

Pushing a crew to extend work schedules will result in performance decrements, no 
matter how dedicated and motivated the personnel may be. 

Difficult Operational Environments 

HSI issues in difficult environments should be analyzed to determine risk areas and to 
identify potential solutions. Night operations are an important example. Dark adaptation 
phenomena can be important for display design. The possible use of image 
intensification or night vision devices by UAV operators should be addressed. Other 
environmental issues should be reviewed, such as extreme temperatures, precipitation, 
and poor visibility (sand, dust, fog). Finally, worst-case scenarios must be entertained, 
such as the requirement to operate in chemical and biological protective gear (MOPP 4). 
The UAV equipment interfaces, training, and procedures must be capable of supporting 
effective operations in all environments. 
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Procedures and Job Performance Aids (JPAs) 

System effectiveness will be improved by the development and application of doctrine, 
standard procedures, teamwork guidelines, and JPAs. Development of these policies, 
procedures, and job aids requires input from expert analysts working with experienced 
operators. HSI experts who are knowledgeable in job design, JPAs, and training should 
be included in this process, along with subject matter experts (SMEs) with relevant 
SUAV experience. 

The Navy has conducted an aggressive trial program in JPA applications wherein every 
job onboard the USS Preble (DDG-88) was provided with a JPA in the form of a 
personal digital assistant (PDA). Initial findings were strongly positive, indicating that the 
JPAs reduced training time and supported job performance improvements (Booz Allen 
Hamiliton, 2002). The same benefits can be enjoyed by the UAV community by 
deploying JPAs. 

Moving Control Platforms 

Controlling UAVs from a moving platform presents system design challenges, whether 
the controller is in the air, on the ground, or aboard ship. Human controllers will be 
confronted with the requirement to maintain spatial orientation and situational awareness 
(SA) of the controlled UAV(s), while simultaneously sensing and understanding the 
dynamics of their own vehicle. For example, De Vries and Jansen (2002) found that 
display of own-vehicle motion reduced the operators’ ability to perceive the direction and 
flight path of a controlled UAV. 

The following key issues should be addressed to evolve the capability to manage one or 
more unmanned vehicles from a moving vehicle: 

• How should spatial information about own-vehicle and controlled vehicle(s) be 
displayed to the UAV operators? 

• Can individual differences in spatial orientation and mental rotation be used as 
criteria for selecting and assigning UAV operators? 

• What types of training and simulation systems are necessary to develop the 
necessary skills to manage complex multivehicle dynamics? 
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Dowell, Shively, and Casey (2003) found that a heading tape display is more effective 
than a compass rose display for supporting UAV operator performance. These findings 
have practical utility, but in the future, this area of research will need to be extended to 
determine how a moving operator (in a HMMVW or an AH-64 Apache, for example) can 
best retain spatial orientation and SA with dynamic displays of multiple vehicle dynamics 
(Durbin, Havir, Kennedy, and Schiller, 2003). 

In addition to the issue of navigation/spatial orientation, moving vehicles present three 
other HSI challenges: 

1) motion sickness; 

2) biodynamic interference with manual control; and 

3) head-mounted display (HMD) bounce. 

Motion sickness occurs in these situations due to a combination of actual motion plus 
“cybersickness” (McCauley and Sharkey, 1992; Morphew, 2003). The visual cues to 
motion, perceived by information displayed on the HMD, differs from the motion 
perceived by the inner ear, setting up a “cue conflict” situation (Reason and Brand, 
1975). 

Biodynamic interference is simply the feed-through to manual control interfaces from the 
shock and vibration of rough terrain, transmitted through the vehicle suspension to the 
operator’s body, arms, and hands. This can be a significant problem, depending on the 
vehicle, the speed, and the terrain. 

HMDs tend to resonate at certain frequencies of vertical axis vibration, which are typical 
in ground vehicles (Sharkey, McCauley, Schwirzke, Casper, and Hennessy, 1995). As 
the HMD “bounces” relative to the face and eyes, degraded visual performance will 
occur. 

These HSI challenges in operating UAVs from moving platforms will depend greatly on 
type of vehicle, the terrain, and vehicle speed. For these reasons, HMDs may not be 
suitable display devices for UAV controllers, especially in moving control platforms. 
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LONGER-TERM PERSPECTIVE 

Despite decades of research and development on UAVs, this technology is still primitive, 
with the possible exception of the more advanced and expensive HALE UAVs. Current 
SUAV technology continues to rely heavily on the human controller acting as a remote 
controller. Given this legacy design concept, the human controller is not capable of 
directly controlling multiple UAVs effectively, especially in challenging conditions. 
Breakthroughs in UAV system capability must come from pushing into new conceptual 
areas such as intelligent, semiautonomous agents of varying size and capability 
operating as a team in response to “plays” called by the human supervisory controller. 

Automation 

Advances in technology are proceeding rapidly in many areas that are relevant to 
unmanned systems such as robotics, semiautonomous systems, biomimetics, and 
agent-based systems. A brief overview of selected technology topics is given in 
Appendix A. The term “autonomy” can be applied to all of these technologies because 
they contribute to a shift away from the human-in-the-loop control that characterizes 
SUAV control systems. 

Automation is likely to play a large role in future UAV systems, but should not be viewed 
as a panacea. The introduction of highly automated systems can cause serious 
difficulties such as mode confusion, “automation surprise,” distrust, complacency, and 
the “out-of-the-loop” operator performance problem (Endsley and Kiris, 1995; 

Sarter, Woods, and Billings, 1997; Sheridan, 1992; Weiner, 1988). 

“Clumsy automation” is the term used to describe automation that makes easy tasks 
easier and difficult tasks more difficult (Weiner, 1988). This is the opposite of the 
intended objective of automation, which is to reduce peak workload (i.e., reduce the 
difficulty of difficult tasks). Woods (1993) summarized studies on automation design this 
way: “New technology introduced for putative benefits in fact introduced new demands 
and complexities into already highly demanding fields of practice.” 

Many automation functions could be applied to SUAVs, ranging from improved 
“autopilot” functions to autonomous, collaborative swarms. The former functions were 
operational 50 years ago; the latter are not yet operational. But, ignoring the technical 
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feasibility for the moment, what design strategy should be adopted for the sequential 
introduction of automation features into a UAV system? 

Human-in-the-loop simulation is an excellent tool for addressing HSI issues prior to final 
design of the technological subsystems. The manned-simulation approach enables the 
tradeoffs to be evaluated between the technical requirements, including automation 
features, and the HSI issues such as operator roles, tasking, manning, workload, user 
interfaces, the probability of operator error, and training requirements. 

Allocation of System Functions to Automation or Humans 

As automation technology evolves, how much control should humans relinquish? What 
role should the human operator play in UAV systems? The answers to these questions 
also may evolve, but these HSI and systems engineering questions need to be 
answered prior to the application of automation “solutions.” 

What functions are good candidates for automation? Frequently, automation is 
advocated for jobs that are “dirty, dull, or dangerous.” Although this saying has good 
alliteration, it is neither an exhaustive list nor a valid set of criteria. Systems engineers 
and HSI specialists have the opportunity to determine in advance the control allocation 
architecture. A process analogous to triage may be warranted, where: 

1 ) certain functions may be reserved for human operators; 

2) other functions are shared, mixed, or adaptively allocated to the best available 
resource, either human or automated; and 

3) some functions may be routinely allocated to the automated subsystems. 

Through the use of simulation, the challenge of designing function allocation can be 
addressed early in system design, prior to engineering or technological determinations 
about how ihe automation features might be implemented. 

Table 2 gives a first approximation of separating top-level (human control), mid-level 
(overlapped, shared, or adaptive control), and lower-level tasks (prime candidates for 
automation). The “top-level” list is intended to endure, no matter how sophisticated the 
technology may become in the future. The mid-level tasks are currently accomplished 
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by humans, but could be allocated, at least some of the time (adaptively), to intelligent 
automation in the future. The lower-level tasks also are frequently accomplished by 
humans, especially during takeoff and landing, but current technology is capable of 
performing these functions. 


Table 2. Example of Function/Task Allocation for SUAV Systems. 


Top-Level Human Tasks 

Mid-Level Shared Tasks 

Lower-Level Automated Tasks 

1. Define Mission Objectives 

1. Airspace Deconfliction; 
“See and Avoid” 

1. Vehicle Control (autopilot 
functions) 

2. Specify/ Assign Team and 
Assets 

2. Terrain Avoidance 

2. Counter Environmental 
Disturbances 

3. Specify Location (region, 
area) 

3. Implement Tactics 

3. Maintain Sensor Lock-on 

4. Specify Time Period 

4. Navigate 

4. Collision Avoidance 
Processing 

5. Determine Tactics 

5. Avoid or Investigate 
Possible Threats 

5. Store or Transmit Data 

6. Determine Contingency 
Plans 

6. Teamwork Dynamics 


7. Initiate Operation 

7. Sensor/Payload Control 

8. Modify Mission Objectives 
and Tactics 

8. Takeoff and Landing 

9. Launch Weapons 

10. Terminate/Recall Mission 




Sheridan (1992; 2002) has provided excellent insights into automation, supervisory 
control, and system design. As mentioned earlier, a common error is to automate 
whatever functions are easy to automate and let the human do the rest. As noted by 
Sheridan (2002): 

From one perspective this dignifies the human contribution. From 
another it may lead to a hodgepodge of partial automation, making the 
remaining human tasks less coherent and more complex than they need 
be, and resulting in overall degradation of system performance. 

Advancements in the human-system engineering processes for developing automation 
functions in manned military aircraft are relevant to UAV systems (Taylor, Bonner, 
Dickson, Howells, Miller, Milton, Pleydell-Pearce, Shadbolt, Tennison, and Whitecross, 
2002). The authors advocate adaptive automation and decision aiding based on 
cognitive systems engineering principles. Communication of intent is important 
because, in a hierarchical tasking system, the intent of the commander flows down 
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through the task hierarchy. The concepts and methods suggested by Taylor et al. 
(2002) were integrated in the Cognitive Cockpit (COGPIT) program of the 
United Kingdom Defence Evaluation Research Agency (DERA). These concepts are 
applicable not only to aircraft interfaces, but to UAV systems, both for user interface 
design of the GCA and for automation system design. 


One approach to defining types and levels of automation has been suggested by 
Parasuraman, Sheridan, and Wickens (2000). This model or framework features the 
selection of a level of automation for each of four stages of human information 
processing, as shown in Figure 4. 

Figure 4. Levels of automation for each of four stages of human information processing 

(from Parasuraman et al., 2000). 


Information Information Decision 

Acquisition Analysis Selection 


Action 
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According to this model, four stages of human information processing can be applied 
both to human information processing and to automated functionality. The design of any 
specific system can apply automation at a selected level in each of the four stages. In 
the example given in Figure 4, System A has less automation than System B and has its 
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highest level of automation in the “Information Acquisition” stage. A medium to high 
level of automation in this first stage would imply support for sensor management and 
perhaps some degree of highlighting, preprocessing, or preliminary object 
detection/recognition aiding. 

Parasuraman et al. (2000) emphasize the importance of testing and evaluating 
preliminary choices of automation functionality. Iterative testing establishes the best 
automation levels for supporting human operator performance. A secondary testing 
criterion would be the reliability of the automation functionality. Reliability of automation 
is important for achieving the trust of the human operator. 


There have been several attempts to define stages or levels of automation. 
Sheridan (1992) provided a definition often levels of automation (see Table 3). 


Table 3. Sheridan’s “Scale of Degrees of Automation” (from Sheridan, 1992). 


1 . 

The computer offers no assistance; human must do it all. 

2. 

The computer offers a complete set of action alternatives, and 

3. 

Narrows the selection down to a few alternatives, or 

4. 

Suggests an alternative, and 

5. 

Executes that suggestion if the human approves, or 

6. 

Allows the human a restricted time to veto before automatic execution, or 

7. 

Executes automatically, then informs the human, or 

8. 

Informs human after execution only if asked, or 

9. 

Informs human after execution if it, the computer, decides to. 

10. 

The computer decides everything and acts autonomously, ignoring the human. 


Some of the HSI challenges in developing effective automation systems in UAVs are to 
develop: 


1 ) criteria for the application of automation features; 

2) a consistent and unambiguous way to inform the user about the status of modes 
and automation features; and 

3) a design that empowers the human user to reacquire control of any automated 
variable upon request. 
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HSI Issues in Automation 

Distrust and Complacency . When automation is introduced, the job of human controllers 
usually is shifted from direct control to supervisory control (Sheridan, 1992; 2002). A 
common reaction to this shift in roles and responsibilities is for the human controllers to 
distrust the automation features. This problem is exacerbated when early versions of an 
automated system are perceived to be unreliable, which can be expected when reliability 
is less than 95%. 

In a recent review of trust in automation, Lee and See (2004) defined trust as “the 
attitude that an agent will help achieve an individual’s goals in a situation characterized 
by uncertainty and vulnerability.” After automation has been integrated into ongoing 
operations the problem of “distrust” is more likely to be replaced by the problem of 
overtrust or complacency. The users (human operators, supervisory controllers) trust 
the automated system too much, leading to complacency in their role as “supervisory 
controller.” 

Lee and See (2004) use the terms “appropriate trust” or “calibrated trust” because 
humans could err in either direction — distrust a reliable automation feature or overtrust 
an unreliable automation feature. Distrust leads to disuse (the capabilities of automation 
are rejected), whereas overtrust leads to complacency and misuse (violation of 
assumptions or inappropriate reliance on automation). 

Design and training approaches are suggested by Lee and See (2004) to achieve 
appropriate trust in automation, including: 

• Design for appropriate trust, not greater trust; 

• Show the past performance of the automation; 

• Show the process and algorithms by revealing sensible intermediate results; 

• Show the purpose and design of the automation in a way that relates to the 
user’s goals; and 

• Train operators about the expected reliability and intended use. 
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Case Example . A case example of overtrust or complacency was observed at 
Camp Roberts as a UAV was returning to the landing strip under automated control of 
altitude and heading. Because no direct, manual control was being exercised and no 
search process was ongoing, the operator(s) were loosely monitoring the video image 
on a small back-up display. The unscheduled descent was not noticed until it was too 
late. Upon impact, the video imagery flow became static, leading one operator to 
exclaim, “I think we may have crashed!” The three other team members met this 
statement with incredulity, until they observed the static video close-up of the ground. 

An intermittent software error was thought to have been the likely source of the problem, 
extending landing gear and cutting power. An alert operator, monitoring the situation 
closely, may have had time to reacquire manual control prior to impact. Overtrust in the 
altitude-hold automation feature may have contributed to the failure to detect the 
unintended descent. 

Human Error in Automated Systems . Reduction of human error often is given as a basis 
for the decision to automate system functions. If the human is removed from the control 
loop, he cannot make an error. Or can he? Early use of automation in the commercial 
airline industry provides lessons that can be applied for UAV control. Because a 
majority of airline crashes were attributed to “human error,” many technologists in the 
1970s believed that automation was the best way to reduce human error. Wiener and 
Curry (1980) analyzed cockpit automation as the “glass cockpit” was introduced into 
commercial aviation in the 1970s. Their analysis did not support the assumption that 
automation would decrease human error. Rather, the authors suggested that 
automation changes the types of errors and, in fact, may create opportunities for new 
types of human-system errors. 

Human error is complex (Reason, 1990). One reason to adopt a systems engineering 
and HSI perspective on the development of automated systems is to identify in advance 
various forms of human error and design the system to prevent them or limit their 
consequences. 

Mode Confusion . The introduction of the “glass cockpit” to commercial aviation in the 
1970s required a transition period during which flight crews had to learn new controls, 
displays, new roles (supervisory control), and a proliferation of computer-controlled 
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automation features. One common problem with the new feature-rich automation 
systems was mode confusion, sometimes resulting in “automation surprise” (Sarter and 
Woods, 1995; Sarter, Woods, and Billings, 1997). 

This problem of mode confusion and automation surprise occurs for one of two reasons: 

1 ) the effect of a specific operator (pilot) input is mode-dependent; or 

2) the operator is surprised at the behavior of the automated system because 
he/she made an incorrect assumption (or had an inadequate mental model) 
about the current mode of operation. 

The likelihood of such an error is more apparent when one understands that one type of 
commercial airline cockpit has 1 1 auto thrust modes, 17 vertical flight path modes, and 
10 lateral flight path modes. This type of feature proliferation and complexity sets the 
stage for human operators to suffer mode confusion, to enter erroneous data, or 
otherwise interact with the automated system in a way that does not achieve the 
intended result. 

Mode confusion is not a problem with current UAVs, but will be arriving soon, as more 
complex automated systems are introduced. If system designers focus on HSI and on 
applying the “lessons learned” from the introduction of automation in commercial aviation 
over 20 years ago, the result will be a more graceful introduction of automated features 
into UAV systems. 

Integrating human controllers into automated systems is a critical research goal for UAV 
systems. The fundamental engineering technologies that enable UAV functionality, such 
as aeronautical systems, propulsion systems, sensors, and communications links, need 
continued development. But, in addition to those fundamental engineering technologies, 
major improvements in SUAV effectiveness and reductions in life cycle cost can be 
obtained from human-centered design of automation features. 
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RESEARCH RECOMMENDATIONS 

Throughout this report, opportunities for research and development have been 
suggested directly or implied. The following list is a compilation of research 
recommendations mentioned in this report, sorted by major categories: 

Automation 

1. Determine the automation features, selection criteria, and training requirements 
that would be needed to support the control of multiple SUAVs by a single 
operator. 

2. Perform research on defining the roles, responsibilities, and tasks for human 
operators as automation technology evolves. 

3. Develop UAV mission management (planner/replanner) systems to reduce 
operator workload. 

4. Develop a search-aid software tool to assist UAV operators to select and 
implement good search tactics for given situations. 

5. Perform research on adaptive automation systems, mixed-initiative systems, and 
collaborative control to determine successful strategies and designs for 
semiautomated UAV systems. 

6. Perform research to establish guidelines for avoiding mode confusion and 
automation surprise in semiautomated UAV systems. 

Manpower, Personnel, and Training 

7. Conduct research and analysis on how to improve the ratio of humans to SUAVs 
without incurring a reduction in system effectiveness. 

8. Reduce the manpower requirements of a SUAV unit by implementing HSI design 
for maintenance and flight operations support. 

9. Develop training materials for the instruction and qualification of image analysts, 
including mission rehearsal capability to detect, classify, and identify objects and 
entities relevant to upcoming mission. 

10. Develop embedded training systems to promote skill acquisition and 
maintenance using the GCS as the training simulator. 

11. Develop personnel selection criteria based on quantitative measures of UAV 
system performance (e.g., probability of detection; time to detection; probability 
of false alarm; spatial orientation; vehicle control; sensor control). 


31 



Human Systems Integration and Automation Issues in SUAVs 


Mishaps and Mishap Reporting 

12. Develop a standardized, DoD-wide reporting system for UAV mishaps. 

13. Reduce SUAV mishap and loss rates by analysis of losses where operator errors 
are cited as a causal or contributing factor. 

HSI and Systems Analysis 

14. Execute a thorough HSI and systems engineering analysis of current TUAV and 
SUAV operations to determine major contributors to success (and failure) of 
current systems (aeronautics, propulsion, sensors, communications, control 
systems, user interfaces, tactics, and utilization of data in the larger command 
and control environment). 

15. Ensure contribution of HSI to the ongoing process of the design and 
development of a common console for UAV control. 

Policy and Procedures 

16. Prepare for 24/7 operations by developing guidelines for managing sleep and 
fatigue through chronohygiene and watch-standing procedures. 

17. Design and train for difficult environmental conditions — night, poor visibility, and 
chemical defense gear. 

Human Factors Engineering of System or Subsystem Design 

18. Define UAV detection hardware and software system requirements by 
establishing criteria (time, contrast, resolution, visual angle, motion, and other 
relevant factors) for human detection and identification of mission-relevant 
objects. 

19. Perform research to determine the best practices for using static and dynamic 
imagery to support human target detection. 

20. Develop hardware and software to stabilize the visual imagery. 

21 . Implement the technology to monitor the coordinates of a camera or other sensor 
aboard SUAVs. 

22. Develop Job Performance Aids (JPAs) and decision aids for UAV operators 

23. Develop design guidelines for displays and other system features that will enable 
UAV operators to perform their tasks in a moving vehicle (air, ground, or surface) 
while maintaining spatial orientation and situation awareness. 
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24. Develop simulation capabilities to enable analysis of UAV effectiveness as a 
function of new technologies, automation, procedures, displays, and other 
system features. 

Research Recommendations from Appendix A 

25. Perform research on intelligent agent systems and multiagent systems as a basis 
for improving SUAV effectiveness while reducing the UAV footprint and operator 
workload. 

26. Perform research on levels of autonomy and semiautonomous systems with 
application to SUAV systems. 

27. Perform research on swarms and stigmergy to determine how to apply them to 
automation in SUAV systems and micro UAV systems with continued focus on 
the human operator role as supervisory controller. 

28. Determine the interchange and mutually beneficial overlap between 
Human Systems Integration and Human Robot Interfaces with reference to 
SUAV applications. 

29. Perform research on how hierarchical control architectures like Playbook might 
be implemented in SUAV systems with one human operator controlling multiple 
SUAVs. 
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APPENDIX A: AUTONOMY TECHNOLOGY 

This overview is intended to identify some of the research issues that are being 
addressed in autonomy and related disciplines that are relevant to future UAV design 
and operations. It is not a comprehensive review of the scientific and technical literature 
in these fields. 

Robotics, Intelligent Agents, and Multiagent Systems 

UAV development can be considered a subfield of robotics and, thus, advances in 
robotics are important to UAV design and performance. Research and development 
programs in robotics are ongoing at leading universities in the U.S., Europe, and Asia. 

Robots (including UAVs) face many of the same challenges that human perceptual 
systems perform so well, such as visual-inertial stabilization and the analysis of whether 
you are moving or the environment is moving (Panerai, Metta, and Sandini, 2000). 

These intersensory processes and feedback loops provide the foundation for exploiting 
maximum visual acuity (gaze stability) and for discriminating between self-motion and 
motion of the visual scene. Compensation for motion of either sort is necessary to 
maintain image stability on the retina (or optic sensors). As mentioned earlier in this 
report, the lack of this capability has been observed to be a problem for small UAVs 
without image-stabilization systems. Image stability for some minimal duration is a 
fundamental requirement in an optical detection system, whether a human analyst or an 
automated agent is processing the image. 

A demonstration of autonomous robotic capability called ’’The Grand Challenge” was 
sponsored by the Defense Advanced Research Project Agency (DARPA) in March 2004. 
The top 15 autonomous ground vehicles participated in a challenge to win a $1 million 
prize by navigating 140 miles through the Mojave Desert from Barstow, California to 
Primm, Nevada. The entrants included teams like Cal Tech, Berkeley, and 
Carnegie-Mellon using significantly customized vehicles built on the chassis of 
Chevy Tahoes, HMMWVs, and other off-road vehicles. Not only did none of the robots 
make it to the destination, only two of the teams made it as far as seven miles. Most of 
the entries failed to traverse more than a few hundred yards. According to one news 
item, “it was a pretty humbling display for unmanned robot vehicle technology.” The 
results underline DARPA’s prescient selection of the event name, “The Grand 
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Challenge.” Obviously, overcoming difficult environmental obstacles is truly a grand 
challenge for autonomous off-road ground systems in 2004. 

The results of the 2004 DARPA Grand Challenge indicate that frequent “reality checks” 
(test and evaluation under operational conditions) are necessary as autonomous 
technologies are developed and before the lives of U.S. military personnel are 
dependent on their success. 

Intelligent agents have the common requirement that they must be goal directed. An 
agent must exhibit purposeful behavior and have an explicit representation of their goal 
(Huber, 1999). Research on multiagent systems (MAS) has commonly led to hierarchies 
of autonomy with various decision-making procedures employed by the agents. MAS 
prototypes have demonstrated capability in simulated adversarial games such as robot 
soccer. Stone (1998) applied machine learning techniques to MAS and demonstrated 
that a group of independent agents can work towards a common goal in a complex, real- 
time, noisy, collaborative, and adversarial environment. 

Research in Sweden has developed an approach for directing a hierarchy of intelligent 
agents in real time. Their approach was to enable control of UAVs by allowing the 
human controller to access an agent at any level of abstraction, as needed. The human 
can acquire control over an agent in a hierarchy, but the agent will be expected to 
continue with other tasks as well as fulfilling the operator’s request (Scerri, Reed, and 
Torne, 1999). This architecture avoids the common problem of agents abandoning 
tasks when interrupted by the human operator. Agents at lower levels of the hierarchy 
negotiate among themselves to decide which will be responsible for a task given by 
either the human operator or a higher-level agent. As in a human organization, 
guidance and requirements can be issued at varying levels of abstraction and carry 
different weights. 

In Australia, researchers have developed autonomous mobile robots with local 
intelligence. The robots do not rely on a Global Positioning System (GPS), but use 
on-board “vision” sensors to perform tasks such as navigation, map generation, and 
coordinated group behavior (Petitt and Braunl, 2003). The authors attribute the success 
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of this research program to the use of a “behavior-based” approach to controlling the 
mobile robot agents, rather than the traditional control algorithm approach. 

Under the DARPA Mixed-Initiative Control of Automa-teams program (MICA) one group 
of researchers applied the “Playbook” concept to enable a single human operator to 
supervise a team of six robots in RoboFlag, a simulation of the child’s game “capture the 
flag” (Parasuraman, Galster, and Miller, 2003). Emulating a child’s game may seem 
trivial until one appreciates that success requires teamwork, perception of the location of 
the high-value target, and tactics for countering opposition offense and defense. These 
capabilities are not unlike a UAV ISR mission. 

Johnson (2003) described the DARPA MICA program as having two approaches to one 
goal — the control of large-scale teams of semiautonomous vehicles by a relatively small 
number of human operators. The two approaches are: 

1 ) autonomous control theoretic techniques; and 

2) mixed-initiative techniques for integrating humans into the control process. 

The first approach is based on a hierarchical command and control structure including: 

• Team composition and tasking; 

• Team dynamics and tactics; and 

• Cooperative path planning. 

The second approach focuses on how to design the system around the potential 
behavior of the human operators: 

• Meaningful cooperation between human decision makers and teams of 
semiautonomous entities; and 

• How is the performance and stability of the system affected when an operator 
can take control at varying levels and times? 

Johnson (2003) makes the important point that “it is rare historically for an R&D program 
to explicitly consider human interaction issues simultaneously with technology 
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development.” The DARPA MICA program objectives established a requirement for 
robotics and control engineers to work collaboratively with cognitive engineers and HSI 
experts to achieve a human-centered design perspective. 

The MICA program was discontinued after approximately two years into the four-year 
plan. According to the coordinator of the MICA program, the progress was apparent 
during the first two years, but it also became clear that both the Unmanned Combat 
Armed Rotorcraft (UCAR) and Joint Unmanned Combat Air Systems (J-UCAS) 
programs would have to pursue developments similar to MICA in multi-UAV control. 
DARPA concluded that further development of this technical topic should be continued 
within the scope of those larger and pragmatically focused programs (Kott, 2004). 

Autonomy 

A key issue for the design and implementation of future UAVs is “levels of autonomy,” 
which can, theoretically, range from teleoperation to fully autonomous systems. 

Yavanai (2003) defined autonomy as, 

an attribute of a system which characterizes its capability to accomplish 
the system’s assigned mission goals without any, or with only minimal, 
abstract level intervention of an external cooperative entity, i.e., a remote 
agent or a remote operator, while the system is operating under 
constraints and under unstructured, unexpected, and dynamic uncertain 
environment as well as under evolving dynamic uncertain scenario 
conditions. 

Neidhoefer and Krishnakumar (2001) developed an intelligent aircraft control 
architecture based on “levels of intelligent control.” They adopted concepts from 
computational intelligence such as neural networks, genetic algorithms, and 
adaptive critics. 

Robotics research programs have wrestled with ways to enable multiple levels of 
autonomy. Studies at the Johnson Space Flight Center have identified the need for 
adjustable autonomy (AA) because humans sometimes need to be involved, not only at 
the highest (decision/deliberative) level, but all the way down to teleoperation of systems 
that were intended to be fully autonomous (Bonasso, 1999). Their experience in 
developing space applications led to the following recommendation: design the control 
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architecture for full autonomy; then relax the autonomy restriction at each level, 
beginning with the highest. 

The Canadian Defence Research and Development group compiled a technology 
summary of autonomous collaborative unmanned vehicles (Bowen and MacKenzie, 
2003). The report concludes that autonomous control is the key to achieving maximum 
UAV utility. This conclusion extends to Unmanned Vehicles (UV) of all kinds — space, 
air, ground, water surface, undersea, and other types of vehicles. Gaps in knowledge 
were identified in the following technology streams: Robotics, Mobility, and Navigation. 

Another approach is to consider an intelligent robot to be capable of acting as a peer of 
the human operator, worthy of exhibiting collaborative control (Fong, Thorpe, and Baur, 
2001). This type of relationship between the human and the robot also is called a 
“mixed-initiative” architecture. 

One objective is to develop a control architecture that will allow a single human operator 
to interact with multiple robots, while maintaining reasonable workload and effectiveness 
(Goodrich, Olsen, Crandall, and Palmer, 2001). This objective can be extended to allow 
multiple human users to manage multiple robots, perhaps of different types, from 
multiple control platforms, some of which may be mobile. 

Swarms and Stigmergy 

Several concepts merge under the heading of “swarms” including biomimetics, 
stigmergy, self-organizing systems, and emergent behavior. These concepts are worthy 
of analysis for their potential application to UAV design and operations. “Biomimetics” is 
the term given to biologically inspired technology (Bar-Cohen and Breazeal, 2003). 
Biomimetic programs seek to simulate or mimic the mobility, intelligent operation, and 
functionality of biological creatures. 

One aspect of biomimetics is the analysis of swarming behavior, which is common in 
nature (e.g., bees, wasps, and ants). Similarly, flocks of birds and schools offish exhibit 
closely coordinated group behaviors (Clough, 2003). 
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According to a NASA/JPL Website rhttp://dsp.ipl.nasa.qov/members/payman/swarm/l . 

Swarm Intelligence (SI) is the property of a system whereby the collective 
behaviors of (unsophisticated) agents interacting locally with their 
environment cause coherent functional global patterns to emerge. SI 
provides a basis by which it is possible to explore collective (or 
distributed) problem solving without centralized control or the provision of 
a global model. 

The rise of agent-based simulation systems is in concert with swarming biomimetics. 

Hoffmeyer (1994) defined a swarm as, “a set of mobile agents which are liable to 
communicate with each other directly or indirectly and which collectively carry out a 
distributed problem solving.” An individual insect has extremely limited intelligence, yet, 
by following simple rules, the emergent behavior of the swarm efficiently gathers food, 
transports/retrieves heavy prey, and defends itself. 

Stigmergy is the use of the environment to communicate. Bonabeau and Theraulaz, 
(2000) discuss how ants leave a trail of pheromones, a form of stigmergy, to forage 
efficiently. For example, an ant leaves a pheromone trail outbound and, after finding 
food, doubles the intensity of the pheromone trail when inbound. Subsequent ants will 
tend to follow the strongest trail. These same rules — the pheromone serves as an 
attractor sign, but has a decay function — can be used computationally to solve difficult 
problems such as the “traveling salesman.” In this problem, a person must find the 
shortest route by which to visit a given number of cities, each only once. This classic 
problem is “devilishly difficult” using traditional computational methods because for just 
15 cities, there are billons of route possibilities (Bonabeau and Theraulaz, 2000). For 
UAV applications, an analogous type of communication among the “swarm” could 
provide information about what areas of a search region have been recently covered. 

“Digital pheromones” have been suggested by Parunak, Purcell, and O’Connell (2002) 
as an effective way to coordinate swarming UAVs. They describe digital pheromones as 
analogous, but better than the use of electrostatic potential fields to control movement. 
New information is quickly integrated into the field, while obsolete information is 
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automatically forgotten through pheromone evaporation. After developing simulations as 
proof of concept, Parunak et al. (2002) concluded that “swarming techniques inspired by 
insect pheromones offer a powerful mechanism for coordinating unmanned vehicles 
such as UAVs.” 

Paul Gaudiano and colleagues at Icosystem Corp. are currently working under contract 
to DARPA to develop control strategies for robot swarms. Gerla and Yi (2004) from 
UCLA are sponsored by the Office of Naval Research (ONR) to investigate team 
multicast communications among autonomous sensor swarms. The Air Force Institute 
of Technology has developed a model of swarm-based, networked, sensor systems and 
espoused measures for the evaluation of swarm performance (Kadrovach and Lamont, 
2002 ). 

One aspect of swarm behavior, seen clearly in flocks of birds and schools offish, is their 
ability to “see and avoid.” That is, they perceive and control their distance to their 
nearest neighbors and adjust altitude/depth and heading accordingly. How can this 
“swarming” emergent behavior contribute to UAV design? One fundamental aspect of 
“see and avoid” is to avoid one’s own teammates. This aspect of the swarming analog is 
entirely relevant to UAV applications. The other aspect of “see and avoid” in UAVs is to 
avoid terrain obstacles and all other aircraft in the National Air Space. It remains to be 
seen whether those issues may be informed by swarming emergent behavior. 

Parunak, Brueckner, and Odell (2003) investigated three main approaches to swarming 
coordination: 

1 ) relationship between individual agents and their group (“roles analysis”); 

2) optimizing systems in light of constraints; and 

3) processes inspired by natural systems (digital pheromones and biomimetics). 

They investigated various instantiations of these approaches by creating software 
models. 

Defence R&D Canada compiled a review of swarming UAVs from a control engineering 
perspective (Kim, Hubbard, and Necsulescu, 2003). The authors describe swarming 
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entities as autonomous units that can gather from different locations, act together, and 
then disperse. Swarming entities are decentralized, tolerant to variances of the units, or 
to addition/deletion of units. 

Kim, et al. (2003) warn that the specificity of fixed-wing UAVs must not be ignored. They 
are not omnidirectional and cannot fly below a certain speed; therefore, group dynamics 
solutions must take into account the aerodynamic limitations of specific fixed-wing UAV 
flight envelopes. 

Researchers and engineers are investigating alternative approaches to control 
trajectories of multiple UAVs, get sensors to the intended search locations, and avoid 
collisions. Vincent and Rubin (2004) from UCLA reviewed the concept of cooperating 
swarms and report on the design and analysis of cooperative search by simulated UAV 
swarms. 

Sigurd and How (2003) reviewed local approaches to swarming based on 
nearest-neighbor interactions or potential fields. They summarize “flocking” approaches 
based on far-field attraction and a near-field repulsion between vehicles, but caution that 
perfect information about the location and dynamics of each vehicle is unlikely for a UAV 
swarm. Sigurd and How (2003) suggest the application of a “total field” approach using 
magnetic fields, and indicate that some biologists believe that similar mechanisms may 
be found in natural systems such as flocks of birds and schools of fish. Their results 
suggest that safe, but aggressive, navigation can be supported by this magnetic dipole 
approach without requiring each vehicle to know the position of any of the other vehicles. 

The Navy program called “SWARM UAV” is only loosely related to the biomimetic 
concepts that inspired the acronym — Smart Warfighting Array of Reconfigurable 
Modules. This program was focused on the aeronautical development of a small, 
inexpensive UAV that could be used for a variety of purposes, including possible swarm 
applications (Castelli and Howe, 1999). This SUAV later became known as Silver Fox. 

Figure 5 provides a schematic contrast of the control links and the associated workload 
or cognitive demand placed on a human controller for conventional control of multiple 
UAVs and for swarm control. 
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Figure 5. Control types of multiple UAVs (the thicker the communicating lines, the heavier 
the workload and cognitive demands on the CGS operators). 

Entirely autonomous, emergent behavior is not a reasonable goal for military UAV 
systems. But, incorporating some aspects of effective swarm behavior in combination 
with top-level human control (see the “Playbook” example below) may prove to be an 
effective approach to achieving a favorable ratio of UAVs to humans and accomplishing 
ISR missions with minimal risk to humans. 

Semiautonomy and the Human-Robot Interface 

In semiautonomous systems, a human controller acts in concert with an intelligent 
system (semiautonomous agent). The level of autonomy may span a large range and it 
may be either constant or adjustable, varying with time, task difficulty, scenario segment, 
or the status/health/workload of the human. Designing the logical and the physical 
interfaces between the human(s) and the intelligent system is an important task. This 
interface is beginning to be known as the Human-Robot Interface (HRI). 

According to Olsen and Wood (2004), HRI differs from traditional computer-human 
interface (CHI) design in two key ways: 

1 ) robots operate in a physical world that is not completely under software control; 
and 

2) the physical environment encountered by the robot imposes its own forces, 
timing and unexpected events that must be dealt with by the HRI system. 
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One common goal of HRI is to enable one human controller to control multiple robots. 
The number of robots that can be operated is called the “fan-out” of a human-robot team 
(Olsen and Wood, 2004). Goodrich, Olsen, Crandall, and Palmer (2001) offer the 
following classification for research on HRI: 

1) autonomous robots; 

2) teleoperation; 

3) adjustable autonomy; 

4) mixed initiatives; and 

5) advanced interfaces. 

Work on teleoperated systems is the most mature (Sheridan, 1992), but a difficult 
obstacle to effective teleoperation is the time delay in communications incurred as a 
function of the distance between the human operator and the robot. This is clearly an 
important issue for some UAV operations. In teleoperated systems, two approaches to 
this problem have been taken — the use of quickened or predictor displays and the 
application of automation features combined with supervisory control. 

Research on the human role in supervising robots or semiautonomous systems has 
revealed both benefits and costs of various approaches and HRI designs. Systems that 
reduce human workload by participating in higher-level decision-making tasks are 
effective only if the automation is entirely reliable, which is a difficult undertaking 
(Parasuraman, Galster, and Miller, 2003). Consequently, the current view is that the 
best approach to HRI and to the interaction between humans and automated systems in 
general, is that the interface should be adjustable or adaptive. Humans should be able 
to delegate tasks as needed, and be provided with feedback information supporting 
supervisory control of those tasks. According to Parasuraman et al. (2003), this is 
equivalent to delegation as practiced in successful human teams. 

The term “Optionally Piloted Vehicle” has been used to convey the concept of adjustable 
autonomy. The human controller has the option to take control of the vehicle, reverting 
from an autonomous or shared responsibility to a Remotely Piloted Vehicle, i.e., fully 
manual. The transition or transfer of control is the challenging part of the system design. 
Designing a control architecture and user interface that enables graceful transitions in 
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control in both directions — to and from human controller(s) to semiautomated agents — is 
an enormous challenge, but one that has substantial potential benefits for UAV 
operations. 

Christopher Miller, in conjunction with Honeywell and SMA Information Flow 
Technologies, seems to have coined the term “Playbook” (Miller, Funk, Goldman, and 
Wu, 2004). Using the sports metaphor, Playbook represents a hierarchical system 
where the human, operating as the quarterback or coach, can “call the play” and the 
autonomous agents will execute their roles in the play. In mixed-initiative systems, 
presumably, one or more of the players could suggest plays to the decision-maker. The 
Playbook concept is consistent with earlier work on dynamic function allocation, also 
called “adaptive automation,” in which human and automated tasks can be allocated 
dynamically, depending on operator choice, workload, fitness, or similar variables 
(Kantowitz and Sorkin, 1987; Morrison and Gluckman, 1994). 

MACBETH is a tactical planning software tool designed for applications in which a 
human user must quickly specify a mission to a team of autonomous agents. MACBETH 
combines hierarchical task network planning with constraint reasoning into a 
mixed-initiative planning system consistent with the “playbook” metaphor (Goldman, 
Haigh, Musliner, and Pelican, 2000). 

The Playbook concept deserves further research and development. It provides an 
excellent conceptual architecture for human control of a multiple-UAV system. 
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